Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only add an explicit dependency on an existing resource when the deployments engine will use the GET response #15693

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jeskew
Copy link
Contributor

@jeskew jeskew commented Nov 27, 2024

Resolves #13674
Resolves #15686

This PR reapplies the change from #15447 now that the bug in indexing expression traversal is fixed.

Microsoft Reviewers: Open in CodeFlow

jeskew added a commit that referenced this pull request Dec 3, 2024
#15580)

Resolves #15513

###### Microsoft Reviewers: [Open in
CodeFlow](https://microsoft.github.io/open-pr/?codeflow=https://github.com/Azure/bicep/pull/15580)

Bicep will normally generate an explicit dependency when one resource
refers to another. For example, if the body of `b` includes a symbolic
reference to `a`, then in the compiled JSON template, the declaration
for `b` will have a `dependsOn` property that includes `a`.

However, if `a` is an `existing` resource and the template is not being
compiled to language version 2.0, then the compiler will "skip over" `a`
and have `b` depend on whatever `a` depends on. For example, for the
following template:

```bicep
resource a 'type@version' existing = {
  name: 'a'
  dependsOn: [
    c
  ]
}

resource b 'type@version' = {
  name: 'b'
  properties: {
    foo: a.properties.bar
  }
}

resource c 'type@version' = {
  name: 'c'
}
```

the non-symbolic name output will have `b` depend on `c`.

#15447 added a couple of scenarios in which Bicep would skip over an
existing resource even if compiling with symbolic name support. This was
done because the ARM backend will perform a `GET` request on any
`existing` resource in the template _unless_ its body properties are
never read and no deployed resource explicitly depends on it. The extra
`GET` requests could sometimes cause template deployments to fail, for
example if the deploying principal had permission to use secrets from a
key vault as part of a deployment but did not have more generic /read
permissions on the vault.

#15447 reused some existing logic for skipping over an intermediate
existing dependency that unfortunately had an underlying bug that
manifested when the skipped over resource was looped and used its loop
iterator to index into the dependency once removed. For example, if we
modify the earlier example slightly:

```bicep
resource a 'type@version' existing = [for i in range(0, 10): {
  name: 'a${i}'
  dependsOn: [
    c[i]
  ]
}]

resource b 'type@version' = {
  name: 'b'
  properties: {
    foo: [for i in range(0, 10): a[i].properties.bar]
  }
}

resource c 'type@version' = [for i in range(0, 10): {
  name: 'c${i}'
}]
```

Then in the compiled output, `b` will have an explicit dependency on
`[resourceId('type', format('c[{0}]', copyIndex()))]`. Because `b` is
not looped, the deployment will fail. Related issues will occur if `b`
indexes into `a` with a more complex expression or if there is an
intervening variable.

This PR updates explicit dependency generation to take all steps between
a depending resource and its dependency into account when generating
index expressions. For example, in the following template:

```bicep
resource vnets 'Microsoft.Network/virtualNetworks@2024-03-01' = [for i in range(0, 2): {
  name: 'vnet${i}'
}]

resource subnets 'Microsoft.Network/virtualNetworks/subnets@2024-03-01' existing = [for j in range(0, 10): {
  parent: vnets[j % 2]
  name: 'subnet'
}]

resource vault 'Microsoft.KeyVault/vaults@2023-07-01' = [for k in range(11, 10): {
  name: 'vault${k}'
  location: resourceGroup().location
  properties: {
    sku: {
      name: 'standard'
      family: 'A'
    }
    tenantId: subscription().tenantId
    networkAcls: {
      virtualNetworkRules: [{
        id: subnetIds[k - 11]
      }]
    }
  }
}]

var subnetIds = [for l in range(20, 10): subnets[l - 20].id]
```

`vault` will depend on `vnets[(range(20, 10)[k - 11] - 20) % 2]`. Prior
to this PR, `vault` will instead depend on `vnets[k % 2]`, which is the
wrong vnet.

This PR does **not** reapply the change from #15447 but only addresses
the issue described above. #15447 is reapplied in #15693
Base automatically changed from jeskew/15513-bis to main December 3, 2024 20:35
Copy link
Contributor

github-actions bot commented Dec 4, 2024

Test this change out locally with the following install scripts (Action run 12163008935)

VSCode
  • Mac/Linux
    bash <(curl -Ls https://aka.ms/bicep/nightly-vsix.sh) --run-id 12163008935
  • Windows
    iex "& { $(irm https://aka.ms/bicep/nightly-vsix.ps1) } -RunId 12163008935"
Azure CLI
  • Mac/Linux
    bash <(curl -Ls https://aka.ms/bicep/nightly-cli.sh) --run-id 12163008935
  • Windows
    iex "& { $(irm https://aka.ms/bicep/nightly-cli.ps1) } -RunId 12163008935"

Copy link
Contributor

github-actions bot commented Dec 4, 2024

Dotnet Test Results

    78 files   -     39      78 suites   - 39   30m 29s ⏱️ - 14m 30s
11 507 tests  -     25  11 507 ✅  -     25  0 💤 ±0  0 ❌ ±0 
26 745 runs   - 13 365  26 745 ✅  - 13 365  0 💤 ±0  0 ❌ ±0 

Results for commit bbf3b71. ± Comparison against base commit 8c7496c.

This pull request removes 1846 and adds 632 tests. Note that renamed tests count towards both.

		nestedProp1: 1
		nestedProp2: 2
		prop1: true
		prop2: false
	1
	2
	\$'")
	prop1: true
	prop2: false
…
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Bicep_compiler_handles_corrupted_extension_package_gracefully (\u001f�\u0008\u0000\u0000\u0000\u0000\u0000\u0000
�Ի
�0\u0014\u0006��>E�\u0003��5\u0007�\u000e��\u0015���*�\u0006��*m��/o:�K�K/��Ɯ\u00039I�C����F'�(�T�|��F-%e�z��\u001a�7�(\u0004�#���'i�(+]�Q���\u0007q��J3\u00132\u0010Bp%A\u0010\u000e�\u0005��ҹ������K�扩ɵ��Mw��=�\u0019�n�\u0017\u001c�r��	\R\u0010�1DU ���(����?�(>.���9�bo��\u000f<�L��8��^��<}\u0000\u000c\u0000\u0000,"'7' is an invalid end of a number. Expected a delimiter. Path: $.INVALID_JSON | LineNumber: 0 | BytePositionInLine: 20.")
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Bicep_compiler_handles_corrupted_extension_package_gracefully (\u001f�\u0008\u0000\u0000\u0000\u0000\u0000\u0000
�Խ
�0\u0010\u0000��>E�\u0003��k�Bw\u0017��\u0007�퉊���P\u0010��t\u0010�\u0016��\u0008�\u001bs\u0007��q!��4\u001b09ԖH�UL�Ш���<o1�\u0011�o\u0014Q��\u0004!�\u000c^I��u���Lq�\u000f�Klܹ��i!\u0004WR\u000b"�\u001f\u0017K\u0016>����
WƝ�s�CC.�V��E_���\u000cz|�7^���?\u0015\u0012�?�K�\u0005g\u000cQ���x�������h_BSA� �Bq�:Z㣹Zx�][\u0010\u0004A0�\u0017{��4\u0000\u000c\u0000\u0000,"Value cannot be null. (Parameter 'source')")
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Bicep_compiler_handles_corrupted_extension_package_gracefully (\u001f�\u0008\u0000\u0000\u0000\u0000\u0000\u0000
��K\u000e�0\u0010\u0006�=\u0005'(3�ґ\u0005{�^�Q\u0012\u001f�\u0018\u001e��xw���\u0005�
�����L:}�2��~��c^�R\u001b6�\u0014s\u0003�h=�>�\u0004\u0005�\u000b\u0008�Ā\u0010Q?�$#ں��\u001fe��~\u0010m#ۜ�<CVJ��&�Ę����\u001d&k\u0018�ls�Zwue��.��PL5�}�`���}�>�\u001f�\u0004j&
��@��|�\u0017�����X{� \u0008�`
O)��\\u0000\u000c\u0000\u0000,"The path: index.json was not found in artifact contents")
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Bicep_compiler_handles_corrupted_extension_package_gracefully (\u001f�\u0008\u0000\u0000\u0000\u0000\u0000\u0000\u0003�Ի
�0\u0014\u0006��>E�\u0003���
\u001d\u0004\u0007+R\u0005�U�
X�U�
\u0005_�t\u0010�\u0016�^\u0004�ɁsB�\u000f���^\u001b���DT	�#�7lI�[�\u001b�+@�\u001b\u0006X	�\u0000�ux��.�(c��At\u000eu�f&$�1F%�\u0004I\u001a\u0008\u001c���;wݑ\u0000�uu	�<15����)Ϻʧ~����������L9�;�r�\u0018f\u0004`)x��Q2����zQ|\n��is���\u0002�
<�L��8��^�/��\u0000\u000c\u0000\u0000,"'7' is an invalid end of a number. Expected a delimiter. Path: $.INVALID_JSON | LineNumber: 0 | BytePositionInLine: 20.")
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Bicep_compiler_handles_corrupted_extension_package_gracefully (\u001f�\u0008\u0000\u0000\u0000\u0000\u0000\u0000\u0003��K
�0\u0010\u0006�=EN���5��ޥW\u0008Z�AS�\u0003\u000b��M\u0017��\u00167�\u0011̷�0̄�\u0017�����\u001d��\u0011�\u000ce��
\u0002����\u0000
2�\u0017`@�X�x?�&#��uuXe�Y?H��kOe�#)��U\u0016�\u0005��lV����!����w�⫛\u0017��CC9�\u0010��ɨ��}kƧ�\u0003H���\u0006R�0�\u001f%��/��?���\u0011{�$I�$�'\u001d*\u0006{\u0000\u000c\u0000\u0000,"The path: index.json was not found in artifact contents")
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Bicep_compiler_handles_corrupted_extension_package_gracefully (\u001f�\u0008\u0000\u0000\u0000\u0000\u0000\u0000\u0003��K
�0\u0010\u0006�=E�\u0001Ҽ��н\u001b��\u0007�툊���P\u0010�n�\u00107-n�\u0010̷��0\u0013��;�l��P[�u�c��F=%e�y�I��~���D	�p3�$\u001d�֙ڏ2E�\u001fėظs\u0001)�B\u0008��bDIŤ�\u000b_��jl�+�N�̡!\u0017{+�x�\u0017���A��������S!��\u0003��ZP�\u0010UI�Ɠ����\u0003G�\u0012�
2\u0007�\u0016�\u0003��\u001a\u001f���s�ق \u0008��\u0000���g\u0000\u000c\u0000\u0000,"Value cannot be null. (Parameter 'source')")
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Repository_not_found_in_registry (ArtifactRegistryAddress { RegistryAddress = mcr.microsoft.com, RepositoryPath = unknown/path/az, ExtensionVersion = 0.0.0-placeholder },Azure.RequestFailedException: The artifact does not exist in the registry.
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAndLayersAsync(IOciArtifactReference artifactReference, ContainerRegistryContentClient client) in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/AzureContainerRegistryManager.cs:line 138
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAndLayersAsync(IOciArtifactReference artifactReference, ContainerRegistryContentClient client) in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/AzureContainerRegistryManager.cs:line 138,[(BCP192, Error, Unable to restore the artifact with reference "br:mcr.microsoft.com/unknown/path/az:0.0.0-placeholder": The artifact does not exist in the registry.)])
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Repository_not_found_in_registry (ArtifactRegistryAddress { RegistryAddress = mcr.microsoft.com, RepositoryPath = unknown/path/az, ExtensionVersion = 0.0.0-placeholder },Azure.RequestFailedException: The artifact does not exist in the registry.
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAndLayersAsync(IOciArtifactReference artifactReference, ContainerRegistryContentClient client) in D:\a\bicep\bicep\src\Bicep.Core\Registry\AzureContainerRegistryManager.cs:line 138
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAndLayersAsync(IOciArtifactReference artifactReference, ContainerRegistryContentClient client) in D:\a\bicep\bicep\src\Bicep.Core\Registry\AzureContainerRegistryManager.cs:line 138,[(BCP192, Error, Unable to restore the artifact with reference "br:mcr.microsoft.com/unknown/path/az:0.0.0-placeholder": The artifact does not exist in the registry.)])
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Repository_not_found_in_registry (ArtifactRegistryAddress { RegistryAddress = unknown.registry.azurecr.io, RepositoryPath = bicep/extensions/az, ExtensionVersion = 0.0.0-placeholder },System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443))
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAndLayersAsync(IOciArtifactReference artifactReference, ContainerRegistryContentClient client) in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/AzureContainerRegistryManager.cs:line 138
   at Bicep.Core.Registry.AzureContainerRegistryManager.<>c__DisplayClass4_0.<<PullArtifactAsync>g__DownloadManifestInternalAsync|0>d.MoveNext() in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/AzureContainerRegistryManager.cs:line 44
--- End of stack trace from previous location ---
   at Bicep.Core.Registry.AzureContainerRegistryManager.PullArtifactAsync(RootConfiguration configuration, IOciArtifactReference artifactReference) in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/AzureContainerRegistryManager.cs:line 51
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAndLayersAsync(IOciArtifactReference artifactReference, ContainerRegistryContentClient client) in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/AzureContainerRegistryManager.cs:line 138
   at Bicep.Core.Registry.AzureContainerRegistryManager.<>c__DisplayClass4_0.<<PullArtifactAsync>g__DownloadManifestInternalAsync|0>d.MoveNext() in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/AzureContainerRegistryManager.cs:line 44
--- End of stack trace from previous location ---
   at Bicep.Core.Registry.AzureContainerRegistryManager.PullArtifactAsync(RootConfiguration configuration, IOciArtifactReference artifactReference) in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/AzureContainerRegistryManager.cs:line 63
   at Bicep.Core.Registry.OciArtifactRegistry.TryRestoreArtifactAsync(RootConfiguration configuration, OciArtifactReference reference) in /home/runner/work/bicep/bicep/src/Bicep.Core/Registry/OciArtifactRegistry.cs:line 499,[(BCP192, Error, Unable to restore the artifact with reference "br:unknown.registry.azurecr.io/bicep/extensions/az:0.0.0-placeholder": Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)))])
Bicep.Core.IntegrationTests.AzTypesViaRegistryTests ‑ Repository_not_found_in_registry (ArtifactRegistryAddress { RegistryAddress = unknown.registry.azurecr.io, RepositoryPath = bicep/extensions/az, ExtensionVersion = 0.0.0-placeholder },System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443))
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAndLayersAsync(IOciArtifactReference artifactReference, ContainerRegistryContentClient client) in D:\a\bicep\bicep\src\Bicep.Core\Registry\AzureContainerRegistryManager.cs:line 138
   at Bicep.Core.Registry.AzureContainerRegistryManager.<>c__DisplayClass4_0.<<PullArtifactAsync>g__DownloadManifestInternalAsync|0>d.MoveNext() in D:\a\bicep\bicep\src\Bicep.Core\Registry\AzureContainerRegistryManager.cs:line 44
--- End of stack trace from previous location ---
   at Bicep.Core.Registry.AzureContainerRegistryManager.PullArtifactAsync(RootConfiguration configuration, IOciArtifactReference artifactReference) in D:\a\bicep\bicep\src\Bicep.Core\Registry\AzureContainerRegistryManager.cs:line 51
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAndLayersAsync(IOciArtifactReference artifactReference, ContainerRegistryContentClient client) in D:\a\bicep\bicep\src\Bicep.Core\Registry\AzureContainerRegistryManager.cs:line 138
   at Bicep.Core.Registry.AzureContainerRegistryManager.<>c__DisplayClass4_0.<<PullArtifactAsync>g__DownloadManifestInternalAsync|0>d.MoveNext() in D:\a\bicep\bicep\src\Bicep.Core\Registry\AzureContainerRegistryManager.cs:line 44
--- End of stack trace from previous location ---
   at Bicep.Core.Registry.AzureContainerRegistryManager.PullArtifactAsync(RootConfiguration configuration, IOciArtifactReference artifactReference) in D:\a\bicep\bicep\src\Bicep.Core\Registry\AzureContainerRegistryManager.cs:line 63
   at Bicep.Core.Registry.OciArtifactRegistry.TryRestoreArtifactAsync(RootConfiguration configuration, OciArtifactReference reference) in D:\a\bicep\bicep\src\Bicep.Core\Registry\OciArtifactRegistry.cs:line 499,[(BCP192, Error, Unable to restore the artifact with reference "br:unknown.registry.azurecr.io/bicep/extensions/az:0.0.0-placeholder": Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)) (No such host is known. (unknown.registry.azurecr.io:443)))])
…

@@ -155,6 +161,65 @@ public override void VisitVariableAccessSyntax(VariableAccessSyntax syntax)
}
}

/// <summary>
/// Determines whether a reference to a resource is weak.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weak

For cases when the index expression is a VariableAccessSyntax, we will return false because we're not sure.

Should we refect it somewhere in the comments that if this function returns true it's a definitive answer, but false is more of a "maybe"?


private bool IsWithinScopeDeclaration(SyntaxBase syntax)
=> TryGetCurrentDeclarationTopLevelProperty(LanguageConstants.ResourceScopePropertyName) is { } nonNull &&
model.Binder.IsDescendant(syntax, nonNull);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IsDescendant

My memory is hazy on this one. Does the binder IsDescendant method account for loops for parent and scope scenarios?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants